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Abstract 


Usability  engineering  are  the  set  of  design  and  development  practices  the  Space  & 
Naval  Warfare  Systems  Center  (SSC  San  Diego)  followed  in  order  to  ensure  that  the 
Tomahawk  Operators  can  succeed  in  using  the  Land  Attack  Combat  Systems  (LACS)  for 
its  intended  purposes.  This  paper  presents  our  approach  of  the  usability  engineering 
activities  and  the  results  from  a  1-year  Fiscal  Year  2003  effort  for  the  development  of  the 
LACS  Human  Computer  Interface  (HCI).  Iterative  usability  evaluations  and  design 
processes  were  conducted  quarterly  on  the  LACS  HCI  in  order  to  develop  the  HCI  that 
best  supported  the  tasks  of  the  Tomahawk  Operators.  Usability  evaluations  consisted 
both  of  Heuristic  Reviews  and  Usability  Testing.  Results  from  these  iterative  evaluations 
were  integrated  into  successive  design  builds  for  further  evaluations. 


Introduction  on  LACS 

Increasingly  new  weapons  and  new  controls  capabilities  and  requirements  will 
continue  to  increase  the  demands  placed  on  Navy  operators.  The  capability  for  small 
team  of  operators  to  plan  for  simultaneous  control  and  launch  of  multiple  weapons  types 
is  highly  desirable.  Designing  an  effective  HCI  that  provides  this  capability,  without 
increasing  current  manning  requirements,  workload  or  situational  awareness  levels  is 
future  challenges  (Kellmeyer,  Lulue,  Osga,  and  Campbell,  2001).  Funded  by  two  Future 
Naval  Capability’s  (FNC)  of  the  Office  of  Naval  Research  (ONR),  SSC  San  Diego  is 
leading  the  work  on  the  development  of  the  LACS  HCI.  This  effort  is  the  ATTD 
Transition  Candidate  for  TTWCS  version  6  that  will  enhance  the  current  Tactical 
Tomahawk  Weapons  Control  Systems  (TTWCS)  version  5  HCI. 

The  LACS  HCI  consists  of  two  displays:  The  Task  Manager  and  TacSit.  The 
Task  Manager  provides  the  TTWCS  team  an  interface  for  supervisory  control  of  all 
Tomahawk  Land  Attack  Missile  taskings.  Supervisory  control  allows  the  team  members 
to  work  in  collaboration  with  automation  as  well  as  each  other.  The  TacSit  provides  the 
situational  awareness  plot  with  enhanced  capability  to  provide  the  TTWCS  operators 
with  increased  situational  awareness  and  understanding  to  quickly  see  the  tactical 
situation. 


Figure  2.0  TacSit  (Task  Management) 


Importance  of  Usability  Engineering  for  the  LACS  HCI 

Employment  of  the  Usability  engineering  process  produces  highly  usable  LACS 
HCIs  that  are  essential  to  increased  productivity  and  timeliness,  reduced  manning, 
reduced  mental  workload,  reduced  training,  reduced  human  error,  and  warfighter  trust.  It 
is  the  systematic  application  of  established  human  factors  engineering  expertise  and 


practices.  Usability  development  practices  offer  a  means  of  quantifying,  designing,  and 
testing  the  system’s  “fit”  to  the  warfighter  and  their  tasks.  Activities  in  this  engineering 
process  include,  but  not  limited  to,  task  analysis,  user  and  environment  profiles,  rapid 
prototyping,  heuristic  reviews,  and  usability  testing. 

Too  often,  many  program  managers  and  developers  misunderstand  usability 
engineering  add  cost  and  development  time  to  the  product’s  development  lifecycle. 
However,  applying  usability  engineering  in  the  product’s  lifecycle  reduces  cost  over  the 
life  of  the  product’s  development,  by  reducing  the  need  to  add  missed  or  fix  unusable 
functionality  later  in  the  development  cycle.  As  part  of  SSC  San  Diego’s  spiral 
development,  Human  Factors  Engineers  apply  the  usability  engineering  process  and 
iteratively  perform  heuristic  evaluations  and  usability  testing  in  the  design  of  the  LACS 
HCI.  These  evaluations  were  conducted  rather  cost  effectively,  but  we  gained  valuable 
insight  on  how  the  warfighters  use  and  perceive  the  LACS  HCI.  This  paper  will  focus  on 
these  usability  evaluations  and  present  the  process  and  results  that  stemmed  from  these 
activities. 


Heuristic  Evaluations  of  the  LACS  HCI 

Heuristic  evaluation  is  the  most  cost-effective  usability  evaluation  of  the  usability 
engineering  process,  intuitive  to  conduct,  and  extremely  effective  in  the  early  phases  of 
the  development  process.  It  is  a  technique  in  which  one  or  more  experts  evaluate  a 
design  according  to  established  usability  principles  (heuristics).  Human  Factors 
Engineers  trained  in  cognition  theory  and  human  factors  principles  review  the  product 
and  judge  the  usability  of  the  product  against  these  heuristics.  Potential  usability  issues 
are  identified  and  then  used  to  derive  recommendations  for  improving  the  HCI.  To 
increase  the  effectiveness  of  the  LACS  HCI  heuristic  evaluation,  multiple  Human  Factors 
Engineers  from  SSC  San  Diego,  NAVAIR  Orlando  Training  Systems  Division,  and 
NSWCDD  independently  performed  heuristic  evaluations  on  the  prototype  interface. 
Studies  have  shown  that  independent  evaluations  conducted  by  three  to  five  Human 
Factors  Engineers  identify  most  of  the  potential  usability  issues  (Nielsen,  1993).  Each 
Human  Factors  Engineer  evaluated  the  interface  independently.  The  results  are  then 
combined  to  analyze  common  and  conflicting  results.  The  evaluations  were  conducted 
quarterly  on  various  versions  and  different  levels  of  fidelity  of  the  HCI.  Depending  on 
the  development  cycle,  the  interface  may  have  been  a  power  point  presentation  or  a  high 
fidelity  running  prototype  with  automation  and  instantaneous  feedback  from  the 
warfighters  action.  Over  the  course  of  the  FY  2003  effort,  a  total  of  over  200  potential 
usability  issues  were  found  on  various  versions  of  the  LACS  HCI.  The  usability 
recommendations  found  from  these  heuristic  evaluations  were  folded  into  subsequent 
version  of  the  HCI  for  usability  testing  or  documented  for  additional  research. 


Usability  Testing 


In  a  well-managed  project,  usability  testing  occurs  at  various  times  throughout  the 
development  lifecycle  to  ensure  the  usability  of  the  product.  Usability  testing  is  a  method 
in  which  real  users  of  the  intended  product  do  real  tasks.  Trained  Human  Factors 
Engineers  analyze  the  data,  diagnose  the  real  problems  and  recommend  changes  to  fix  the 
problems.  Throughout  FY  03,  for  the  purpose  of  evaluating  alternative  design  concepts, 
usability  tests  were  conducted  quarterly  on  initial  LACS  HCI  designs.  Usability  tests 
were  conducted  quarterly  at  SSC  San  Diego  and  NSWCDD.  A  total  of  34  fleet  personnel 
from  various  facilities  acted  as  participants.  The  participants  include:  FCTCPAC, 
FCTCLANT,  USS  Stethem  DDG,  USS  Winston  Churchill  DDG-81,  and  COMSECFLT. 
Participants  were  recruited  from  various  locations  to  provide  a  wider  range  of  feedback. 
This  was  also  to  account  for  differing  opinions  and  operating  procedures  from  the  various 
locations. 


Figure  3.0  Participant  in  a  usability  study 


Methodology 

There  are  different  types  of  usability  tests  conducted  on  the  product  at  various 
stages  of  the  development  phase.  During  FY  03,  the  type  of  methodology  conducted  was 
an  Exploratory  Usability  Test.  An  Exploratory  usability  test  is  a  method  best  used  to 
determine  or  explore  the  efficiency  and  effectiveness  of  the  design  concepts,  also  known 
as  the  user’s  conceptual  or  mental  model.  In  this  type  of  usability  test,  the  participants 
are  solicited  for  their  ideas  of  the  design  concepts  presented  to  them  and  how  to  improve 
any  confusing  areas.  There  is  much  discussion  between  the  Test  Administrator  and  the 
participants  since  the  goal  is  to  understand  their  thought  processes  and  for  the  participants 
to  answer  questions  posed  to  them  by  the  Test  Administrator.  As  the  HCI  matures  and  as 
the  development  phase  moves  from  conceptual  to  design,  the  type  of  usability  tests  will 
move  from  qualitative  feedback  to  quantitative  performance  metrics,  also  known  as 
Validation  testing. 


Low  fidelity  testing  with  power  point  presentations  was  conducted  early  in  the 
development  cycle  to  explore  the  operator’s  mental  model  and  design  concepts.  High 
fidelity  usability  testing  was  conducted  with  simulated  prototypes  to  analyze  how  well 
the  warfighters  can  perform  full-blown  tasks.  A  total  of  6  usability  tests  were  conducted 
in  FY  03.  Over  300  usability  issues  were  found  and  recommendations  provided  to 
improve  the  LACS  HCI.  As  with  the  heuristic  evaluations,  recommendations  were 
folded  into  the  subsequent  versions  of  the  HCI  for  further  testing. 

For  each  usability  test,  5-7  participants  participated  in  the  usability  test  at  SSC 
SD  and  NSWCDD.  Participants  were  only  required  to  have  familiarity  with  the  subject 
matter.  The  evaluations  used  a  scenario  that  required  the  warfighters  to  complete  a 
TTWCS  taskings.  The  warfighters  were  allowed  to  freely  navigate  the  HCI.  This 
allowed  the  opportunity  to  observe  the  intuitiveness  of  the  interface  by  using  a  discovery 
type  method,  as  well  as  the  subsequent  leamability  of  the  interface.  As  the  participants 
moved  through  the  various  displays,  the  Test  Administrator  asked  various  questions 
regarding  the  usability  of  the  displays.  These  questions  were  designed  to  elicit 
information  on  the  intuitiveness,  effectiveness  and  efficiency  of  the  HCI  and  whether  the 
information  presented  supports  their  taskings.  The  administrator  recorded  if  the  operator 
took  the  expected  actions  or  gave  the  expected  answer.  The  administrator  and  data 
logger  also  recorded  any  observations  or  operator  comments. 


Results 


Results  from  FY  03 ’s  usability  study  indicated  the  LACS  HCI  effectively 
supports  the  warfighter’s  tasking.  Effectiveness  was  measured  as  whether  the 
information  provided  the  needed  information  to  complete  their  taskings,  whether  the 
warfighters  launched  the  missiles  on  time,  and  anecdotal  feedback  that  one  warfighter 
can  accomplish  the  taskings  of  a  team  of  Tomahawk  operators.  Other  anecdotal  feedback 
states  the  warfighter’s  approval  of  the  LACS  HCI  over  the  current  HCI.  Lastly, 
additional  research  questions  were  raised  and  are  the  focus  of  FY  04  activities. 
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to  Human  Centered  Design, 
Human  Systems  Integration  and 
Optimal  Manning  . . 
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Why  is  Usability  Important? 

■  Systems  need  to  be  “user-friendly”  to 
increase: 

□  User  efficiency 

□  Productivity  and  Timeliness 

□  Situational  Awareness 

□  User  trust 

■  Workload  reduction 

■  Training  reduction 

■  Can  determine  success  or  failure  of  a 
system 


Fleet  Buy-in! 
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Quarterly  Usability  Evaluations 


■  Focus  of  User  Taskings  Per  Quarter 

□  Q1  -  Call  For  Fire  (CFF),  Mission  Data  Update  (MDU),  and  a 
prepare  pooled  missiles  task 

□  Q2  -  In-Flight  Missile  Health  and  Status  monitoring  screen 

□  Q3  -  Post  Launch  Monitor  Phase 

□  Q4  -  Year-end  Operability  Test 

■  HCI  constantly  evaluated  every  quarter 

■  Improvements  are  made  and  re-tested  the 
following  quarter 

■  Validate  design  changes 


10 


Usability  Engineering  for  the  LACS  HCI 


SPA  WA  R 
Systems  Center 
San  Diego 


Usability  Evaluation  Methodology 

■  Heuristic  Evaluation  (HE) 

□  HSI  engineers  individually  evaluate  the  RPT  against 
a  set  of  usability  criteria  (called  heuristics). 


■  Usability  Testing  (UT) 

□  An  inspection  method  of  usability  evaluations. 
Includes  formal  testing  with  fleet  participants. 
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Types  of  Usability  Studies 


1  .Exploratory 

Explore  concept  of 
design  &  “user’s 
mental  model” 


Paper  prototype, 
Power  Point,  or 
Simulated  system 


2.Assessment 

How  well  can  user 
perform  full-blown 
realistic  task 


Simulated  system 


Development  Phase 
Concept  Design 


3.  Validation 

Verification  of 
product’s  usability, 
compares  to 
established 
benchmark 

Simulated  or  Real 
system 


4.  Comparison 

Compare  to 
alternative  design 


Paper,  Power  Point, 
Simulated  or  Real 
System 


Release 


Rubin,  1994,  Handbook  of  Usability  Testing 
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Heuristic  Evaluation  Process 

■  Compare  the  LACS  interface  to  established 
usability  criteria  (“heuristics”) 

■  Conducted  by  HSI  Engineers  from  SSC- 
SD,  NSWCDD,  and  NAVAIR  Orlando 
independent  evaluated  the  LACS  interface 

■  Report  with  prioritized  usability  issues 

■  Over  200  improvement  recommendations 

□  Many  implemented  and  others  require  additional  research 
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Heuristics 


1.  Visibility  of  system  status 

2.  Match  between  system  and  the  real  world 

3.  User  control  and  freedom 

4.  Consistency  and  standards 

5.  Error  prevention 

6.  Recognition  rather  than  recall 

7.  Flexibility  and  efficiency  of  use 

8.  Aesthetic  and  minimalist  design 

9.  Help  users  recognize,  diagnose,  and  recover  from 
errors 

10.  Help  and  documentation 


Nielsen,  J.  (1994).  Heuristic  evaluation.  In  Nielsen,  J.,  and  Mack,  R.L. 
(Eds.),  Usability  Inspection  Methods.  John  Wiley  &  Sons,  New  York, 
NY. 
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Usability  Engineering  for  the  LACS  HCI 


Usability  Testing  Process 

Participants  are  real  users 

Total  of  46  fleet  participants  individually  tested.  Include  participants 
from  FCTCPAC,  FCTCLANT,  USS  Stethem  DDG-63,  USS  Winston 
Churchill  DDG-81,  COMSECFLT 


■  Used  real  operational  scenarios 

Only  1 5  minutes  spent  explaining  the  scenario  and  minimum  training 
on  layout,  symbology  and  color-coding 


■  Observed  and  recorded  participants  actions  and 
comments 


■  Conducted  low-fidelity  usability  tests  using  paper 
prototypes  to  explore  design  concepts  and 
understand  user  needs 


■  Conducted  high-fidelity  usability  tests  using  working 
prototypes  to  assess  how  well  the  sailors  performed 
realistic  tasks  and  to  verify  interface  usability 
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Video  of  Usability  Tests 
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Usability  Engineering  for  the  LACS  HCI 


Usability  Evaluation  Findings 

9  heuristic  evaluation  reports 

□  Tables  summarized  issue,  location,  heuristic  violated,  recommendation,  and 
priority 

6  usability  testing  reports 

Tables  included  summary  of  design  recommendations,  and  relevant  operator 
comments 


Table  1.  Post  Launch  Monitor  Interface  Design  Recommendations 

Rec  U 

Design  Recommendation 

Comment  # 
Referenced 

Missile  Timelines 

1 

Recommend  implementing  a  click  and  drag  capability  on  the  timebars  (like  the  hand  in 
Acrobat  with  pdf  files). 

A1:9 

2 

Recommend  duplicating  the  minute  header  at  the  bottom  of  the  screen.  It  is  tough  to 
lineup  timelines  that  are  farther  down  on  the  page  with  the  timescale  at  the  top. 

A1:9 

3 

Recommend  that  the  missile  timeline  still  show  flex  options  even  though  they  have  been 
passed  and  were  not  selected.  Maybe  gray  it  out  and  put  a  black  "X"  through  it  to  denote 
that  this  flex  was  missed  or  not  selected. 

A1:10 

4 

Recommend  that  the  new  timeline/aimpoint  not  appear  on  the  timeline  until  the 
transmission  has  finished  processing  or .  .  . 

Alternate  Recommendation:  For  any  option  of  redirecting  an  in-flight  missile,  instead  of 
updating  the  original  timeline  after  processing  a  request,  have  a  new  branch  split  off  below 
the  original  timeline.  This  will  help  show  the  exact  point  in  which  the  missile  redirects  and, 
most  importantly,  it  will  allow  for  quick  and  easy  comparison  of  the  new  branch  and  the 
original  branch. 

A1:10 

5 

Make  the  symbology  clearer  so  the  operator  can  more  easily  determine  when  the  last 
possible  second  is  to  make  a  retargeting  or  flex  decision.  Is  it  the  beginning  of  the  tangent 
line? 

A1:10 
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Usability  Engineering  for  the  LACS  HCI 


Usability  Testing  Results 


Over  300  recommendations  for  new  design  ideas  and  improvements 

Implemented  design  changes  were  rolled  into  future  builds  for  validation 
testin 


Version  1.0 


Version  3.0 
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Usability  Engineering  for  the  LACS  HCI 


Usability  Testing  Results 


1 .  Indicated  interface  effectively  supports  user  taskings 

Provided  needed  information  to  complete  their  taskings 
Many  operators  launched  missiles  on  time 

One  operator  can  accomplish  same  taskings  than  that  of  a  team  of 
Tomahawk  operators 

2.  Met  operators’  approval 

“I  like  this,  this  is  sweet.” 

“Still  impressed.  Impressed  every  time  we  come  out  here.” 

“Like  the  displays,  the  layouts,  the  colors.” 

“A  lot  of  data,  but  it’s  not  too  much,  especially  if  there’s  only  1  operator. 
Everything  is  covered.  Looks  good  to  me.” 

3.  Raised  additional  research  questions 
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HCI  Improvements 


■  Better  understanding  of  operator’s  mental  model,  fleet 
CONOPS,  and  tasks 

■  Improved  color  coding  scheme 

■  Provided  better  feedback  as  to  what  LACS  is  completing 
and  what  taskings  operator  needs  to  accomplish 

■  Provided  needed  information  and  better  organization  of 
information  to  support  operator  tasking 

■  Improved  phraseology 

■  Improved  navigation  and  less  searching 

Provided  needed  information  at  a  higher  level  and  detailed  information 
in  a  lower  level 

□  Decreased  number  of  drill  down  interfaces 

■  Provide  improved  attention  management 
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Usability  Engineering  for  the  LACS  HCI 


Example  of  Improvements 


1 .  Improved  color-coding 


2.  Provided  better  feedback  as  to  what 
LACS  is  completing  and  what  taskings 
operator  needs  to  accomplish 
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Usability  Engineering  for  the  LACS  HCI 


Example  of  Improvements 


Provided  needed  information  and  better  organization  of  information  to  support  operator  tasking 
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Thank  you  for  your  time 


Any  Questions? 
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